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Vast numbers of adults in our changing society need specialized training to perfonn well on the 
job. Yet, that training often eludes people who live in rural areas, for they often have difficulty finding 
opportunities to receive further professional development. In an attempt to improve services to rural 
populations, distance education has become an increasingly popular form of continued training to adult 
audiences. The growing emphasis on distance education recognizes the needs of rural professionals who 
may not have the time or resources to travel to more central training sites. 

Like some other professions, the increasing workplace demands on social workers require a feasible 
solution to providing on-the-job training for those who reside in both urban and rural areas. Social workers 
face a difficult situation; they need to know more information to keep up with changing policies and 
provide quality services, while at the same time increased responsibilities make it more difficult to leave the 
office for training or education. While distance learning opportunities have multiplied greatly in the past 
several years, tlie lack of relevant course work and access to the technology necessary to participate in these 
opportunities may prohibit the wide use of distance education for rural social workers. 

There is a critical need to provide carefully designed training to meet the challenges rural social 
workers face, yet limited resources for staff development within the social work agencies create a difficult 
situation in meeting this training need. The consultation and assessment process for planning staff training 
becomes vital in this situation because any training offered must wisely utilize the limited resources by 
concentrating them on the most essential needs. Schoenmaker (1993) addresses this phenomenon and 
reveals a trend toward better communication between trainees and training developers in order to validate 
design decisions prior to developing the training materials. The training team in this study is practicing 
such communication. 

Following a needs assessment, multimediated training for social work professionals was planned. 
The first of ten modules was designed, developed, and beta tested prior to field placement. 

This paper reports results of a beta test specifically aimed at checking the perceived quality of a 
multimedia lesson prior to its release to a field test situation. It represents one step within the process of 
maintaining communication between the trainees and the training developers. In this case, die trainees are 
social workers who live in rural areas and the developers are a team of people with various professional 
skills. 

BACKGROUND 

Collaboration between two major institutions in a rural midwestem state identified specific 
training needs for child welfare workers and a five-year plan to meet them. The State Department of Social 
and Rehabilitation Services (SRS) contracted with the a major state univereity to develop a series of ten 
multimedia lessons (modules) for child welfare workers that would emphasize the educational needs of 
workers in tiie rural parts of the state. 

The first step of the learning process assessed the workers' perceptions of their needs. The second 
step developed a plan to meet those neids. The third step involved design and development of educational 
modules using multimedia (videodisc, computer, workbook) for the delivery. This paper reports on the 
beta test phase of the first module of the project, which was intended to help guide the project toward 
improvement prior to releasing the module as a finished product. The beta test investigated user opinions 
about the content, instructional design, technical integrity, case of use, and general quality of the module. 
Because people's opinions about self-efficacy, ease of use, and quality can influence their attitudes toward 
the instruction and in turn, influence performance and implementation patterns (Fullan, 1982; Klein, 
Knupfer, & Crooks, 1993; Rogers, 1983; Soltani, 1995), it is important that designers consider learners' 
opinions about the quality of material in terms of its usability and perceived value. 

This program was designed so learners could set the pace of the lesson, and work through it at a 
rate commensurate with their available time, ability, and motivation. The learners have the option to start 
and stop at any time, and to repeat portions when necessary or desired. 

The underlying rationale for self-pacing is that individuals learn at different rates. One claim made 
in favor of using technology for instructional is the ability of technology to adapt instruction to individual 
differences thereby gaining efficiency while at the same time promoting achievement Semb et al (1993). 
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Research clearly demonstrates that a variety of self-paced formats such as computer-based 
instruction (Bangert-Drowns, Kulik. & Kulik. 1985; Kulik & Kulik, 1987, 1989; Kulik & Bangert- 
Drowns, 1985; Orlansky & String, 1981), Bloom's Learning for Mastery (Kulik, Kulik & Bangert-Drowns, 
1989), and Keller's Personalized Systems of Instruction (Kulik, Jaska, & Kulik, 1978) engender more 
positive attitudes about learning than conventional instruction. In addition, they are more effective, 
efficient, and preferred by the learners. (Soltani, 1995) 

Formative Evaluation 

It is helpful to think of formative evaluation as an explicit stance throughout the design process in 
which one collects data and uses them to inform the design procedures. WTiile the project is still fluid, the 
beta test serves as part of the formative evaluation. The results of the beta test will be applied to design 
adjustments in this first module and the remaining nine modules. 

Several researchers have developed a basic set of categories by which instructional software can be 
evaluated. For example, Morrison, (1987) placed maximum emphasis on the facet of interactivity which 
he defined as 'the learner in conversation with himself over the material to be learned'. He addressed the idea 
of adaptability where the program gives control back to the learner. Morrison claimed that well-designed 
products offer users an opportunity to find individual pathways through material and encourage learners to 
develop higher cognitive skills of self-assessment and evaluation of their own learning style and patterns. 

Plowman(1989) explored such facets of learning design in as it specifically applies to interactive 
video. Important categories for evaluation which emerged include considerations of navigation and mobility 
through the software, physical aspects of human-computer interaction, screen presentation, quality of learner 
interactions, consistency of appearance and functionality, visual clarity, legibility, access to support in the 
form of 'help' and 'next' options, availability of a bookmark facility, user support tools like a glossary or 
other reference items, the style of user interface which might embody windows or other devices to maximize 
the amount of easily available information. The design of this project considered all of those factors, plus 
some additional ones. 

Instructional Courseware Design 

A courseware production system has been defined as the entire set of activities beginning with 
vague ideas or concepts to be taught and ending with finished materials for the users (Karrer, 1987). A 
production strategy means the combination of resources to set up and run a production process of 
courseware development; it must address the questions of manpower, required skills, and tools for 
development (Karrer, 1987). 

Many authors have called for a team approach to courseware development. Ally (1985) suggested a 
five-member courseware design team which included a manager, instructional designer, computer specialist, 
communication specialist, and content specialist. Collis and Gore (1987) called for a collaborative model 
which included in its design a rigorous system which enabled team members to check and re-check work 
done by themselves and others by assessing congruency at several points in die development process. 
Faiola (1989) described the traditional team with one more perjon, the graphic or visual specialist Because 
assessment is such an important part of the process Collis and Gore suggest placing in the collaborative 
model a step for field testing prototypes. 

Because of the specific nature of the profession, social workers can have a tremendous impact upon 
human liver at particularly vulnerable times. Therefore, it is important that they receive ongoing education 
that will enhance their practical, applied skill concerning human services. Reay (1986) argu-es in favor of 
changing the focus of social work training from content issues to process issues. This could be done by 
limiting the theoretical information and choices, and rigorously encouraging students to master conditioned 
response (CR) learning and progress to understanding the learning process and applying the theory. Using 
this approach of embedding theory in practice, presents on an opportunity for the combination of modern 
technology and adult methods of learning to help students master the process of systematically applying 
theory to practice. 

General psychological research and theory yield fundamental, widely generalized principles 
concerning how individuals think and leani. General psychological principles are largely media-independent. 
Psychological constructs such as schemata (Norman, Gentner, & Stevens, 1976; Spiro & Anderson, 1981); 
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meaning (Bandura, 1971); elaboration (Wagner & Rohwer, 1981); and situated cognition (Brown & Duguid, 
1993; Suchman, 1991) emphasize how individuals organize and retrieve knowledge and establish meaning, 
and hence have far reaching implications for the design of learning systems that will allow the application 
of theory to practice. 

Motivational Adult Instruction 

Designing instruction for adult learners must take in those special considerations that are a major 
factor in adult learning. Adults have specific and unique motivational needs. The motivational aspect of tiie 
design can increase the learners efforts and attract learners to tiie instructional content and methods. (Keller, 
1983) . 

The motivational instruction can increase the amount of time and effort the learner will put into 
the task. Keller (1987), Keller and Suzuki (1988) as well as Keller and Kopp (1987) identify four categories 
of motivating strategics in learning situations, which include: attention, relevance (Keller's ARCS Model), 
confidence, and satisfaction. All of these strategies were incorporated into the design of this program. The 
needs of the learner were assessed and a design team addressed each aspect as follows:. 

Attention . Content related anecdotes, case studies, and biographies are appropriate along with 
animation and sound. (Keller, 1987) The specific content developed by tiie subject matter experts contains 
all the current topics and issues pertinent to the social workers. The appropriate designs by the graphic 
artist take into consideration what is appropriate for tiie user/client 

Relevance . Showing how the instructional materials relate to specific and current needs. How 
the instruction and information relates to ones job, and to real life. 

Confidence . Giving the learner thr. locus of control. Throughout tiic program, the user has 
options, the ability to make choices and the freedom of navigation. 

Satisfaction . Using skills learned in simulated situations Keller & Kopp (1987). The 
interactive video component takes the learner into environments which in effect lets you think while seeing 
and doing. 

Adult professionals typically need to improve work performance. The computer-based interactive 
videodisc training program, has the potential to assist in learning and thus is very motivational. 

Continuing Education 

Computer-based simulations, especially those incorporating interactive video capabilities, have 
been demonstrated as effective instructional methods in medical, industrial, military and business context 
(Reeves, 1988). One positive aspect concerning this formative evaluation is that the module was 
previously approved for Uiree continuing education credits. 

Individualized Learning Process 

According to Gotz, (1990), tiie learner can designate the time and place of learning. Advanced 
programs offer different paths of instruction in tiic pr>cessing of instructional material. The amount of 
material can be selected according to tiie skills and qualifications of the learner. The learner can call up help 
programs and systematically prepare for tests through simulations and visual instruction. A knowledge 
control of or classification of tests results can be integrated into tiie program and the learning sequences can 
be tepeated as often as needed. 

Distance Education 

Distance Education, although long established, h currcnUy in a stage ov rapid evolution. The 
most fundamental motivation for distance education, that of reaching learners who are geographically 
hindered from meeting in a traditional setting, has been expanded to include motivations such as time 
tailoring and flexibility of learner characteristics. At the same time, advances in the technological media are 
adding new dimensions to distance education. In particular, the contribution of 'electronic distance 
education* saategies with respect to the problems of inequities in opportunity for isolated learners or 
learners with special needs is becoming increasingly recognized (Stubbs & Bumham, 1990). 
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Social and Rehabilitative Services was especially supportive of the project because of the large 
number of social workers from -ural areas that would be trained.. They need to receive comparable services 
and have equity of access. The computer-based interactive videodisc program is meeting that need. 

Mecklenburgcr (1990) stated that an advantage of multimedia and the capabilities it adds to distance 
learning is that learners can learn wherever they are. One of the advantages of diis is anytime, anywhere 
learning. 

Effects of Multimedia 

The popularity of electronic learning, especially interactive muldmedia, centers around the 
potentials of the media. Kelly (1990) cited four such potential: 

1. The new media provides rich opportunities to learn. The combination of sight, sound, and interactivity 
provides opportunities to learn new things without ever leaving the classroom. 

2. The media also provides the unexpected benefit of increasing opportunities for teaching people to work 

in teams and honing other interpersonal skills. 

3. The technologies can make it easier to meet the varying needs of individual learners by being able to : 
diagnose what the learner knows; identify any learning difficulties the learner might have; and then 
attend to those needs. Also, the technologies enable the learners to go at their own pace. 

4. The new technologies can provide new tools to assessment of skills. 

Research provides ample evidence for the power of visuals and interactive multimedia to enhance 
learning (Carlson & Falk 1990-91; Dwyer 1982, 1978; Knupfer & Clark, 1992). This multimedia module 
had the potential to offer visually rich instruction. 

RESEARCH QUESTIONS 
Several research questions were of interest. The research questions of particular interest in this 
formative evaluation were: 

1. Do learners like the module? 

2. Do learners believe that the lesson is realistic, valuable, and relevant? 

3. Do learners believe that the multimedia format works well for delivery of diis lesson? 

4. Do learners believe the video sequences are valuable? 

5. Can learners navigate easily within the lesson? 

6. Are learners likely to use the extra features, such as the glossary or content map? 

7. Does the module function properly, including the bookmark feature? 

8. What strengths, weaknesses, and suggestions do the learners have? 

9 How much time does it take to complete the module? 

10. Are tiiere differences in the way social workers and instructional designers rate the module? 

METHODOLOGY 

Subjects 

Subjects for this study included twenty people; 1 1 were social work graduate students or 
professionals, and 9 were graduate students of educational technology. The social work subjects were 
selected because they would be typical of the group of learners for whom the instruction was designed. 
They likely would represent about the same technical skill level and would have background knowledge 
necessary to judge the merit of the content and presentation style of the training. Tne educational 
technology students were typical of instructional designers who would be responsible for developing similar 
multimediated lessons. 
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Materials 

Subjects in this study used the multimediatcd lesson titled "Child Development" which is the first 
module of the Building Family Foundations series. In addition, they completed a 43-itcm questionnaire. 

The multimediatcd lesson was based on an IBM 486 computer system with a Pioneer videodisc 
player. It used Toolbook as the authoring system. ^ 

A set of written materials composed a workbook that accompanied the electronic part of the lesson. 
The workbook was designed as a reference that social workers could keep closc-at-hand and xerox as needed 
for doing their job independently or working with clients. The workbook contained instructions for starting 
the lesson, and any text-based information that a social worker might like quick access to, especially when 
working with a client. For example, the workbook contained a glossary of terms» informational articles, 
references that a social work might like to refer to, and forms that could be duplicated and used on the job or 
distributed to clients. 

The survey instrument used to collect the data contained 29 items based upon a four-point Likert- 
type scale, 10 open-ended questions, and 4 rank-order questions asking what mode of delivery subjects would 
prefer for future lessons. 

Procedure 

Each subject was given a brief introduction to the equipment and asked to complete a learning 
module independently and in its entirety. The instructions included a request for each subject to try to use 
the bookmark feature one time; other than that, no specific requests were made concerning the process of 
module completion. 

Immediately after completing the module, each subject filled out a questionnaire about the 
experience. The completed questionnaire was collected before the subject left the site. Summary statistics 
were gathered and changes recommended based upon the beta test 



Data Analysis 

The data were analyzed using summary statistics, chi square analysis, and qualitative analysis. The 
structured questions combined with the open-ended questions provided an opportunity to gather the kinds of 
information that we anticipated was important as well as the spontaneous reactions of the subjects. 

RESULTS AND DISCUSSION 
The subjects who used the multimediated lesson represented two occupations and thus, two points 
of view; those knowledgeable in the social work content and those knowledgeable about interactive 
courseware design. The social work subjects (SW) were likely to provide information about such things as 
content accuracy, flow, relevancy, and presentation style, wliile the instructional design (ID) subjects were 
likely to react to such things as human interface design principles, graphic design and use, navigation, and 
other features related to instructional design. 

One potential area of difficulty was experience with this type of mediated instruction. Of the 11 
SW subjects, 4 had used a multimediated lesson before while 7 had not, and of the 9 ID subjects, 8 had used 
a multimediated lesson before while 1 had seen multimedia but had not actually experienced using it. Ages 
ranged from 2 1 through 47 years, with an average SW age of 27, an average ID age of 37, and an average of 
31 years overall. 

Realism, Valuable, aud Format 

The overall reaction of both groups was that this kind of program is valuable for learning this 
type of content and it is especially useful in the case of a distant audience, such as rural social workers. The 
S W subjects were relatively new users of technology and their reactions displayed overall enthusiasm of 
having a well-organized instructional program that offered them audio, video, dynamic graphics, and text. 
Approximately 90 percent of the SW subjects and 100 percent of the ID subjects agreed or strongly agreed 
that the content was accurate, while 100 percent SW and 100 percent ED subjects agreed or strongly agreed 
that the content was realistic. 
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Concerning relevancy, 82 percent of the SW subjects and 78 percent of the ID subjects believed 
that the content was relevant to their needs, while 90 percent of the SW and 89 percent of the ID subjects 
believed that the multimedia format presented a convenient way to learn the material. Even though the ID 
subjects were not studying social work, their notation of relevancy would reflect the generic nature of 
interest in child development. The module was designed to help social workers recognize children who 
should be referred to a specialist for evaluation and potential help. Certainly any parent or person who 
works with children in any way would have some interest in the topic. . 

Thirty-six percent of the SW group and 72 percent ID group believed the content needed more 
depth, while 91 percent SW and 100 percent ID agreed or strongly agreed that the module content was clear. 
Virtually all of the subjects believed that the lesson contained valuable information and that it was valuable 
learning experience for them to use this form of instruction. 

All of the SW subjects and 90 percent of the ID subjects believed that this style of lesson was a 
good way to learn the content, while 72 percent S W and 87 percent ID were satisfied with their own 
performance. 

Video 

One area of potential strength for this lesson was the video component. Approximately 80 percent 
SW and 66 percent ID agreed or strongly agreed that the video information helped them to understand to 
concepts. None of the SW subjects believed that the video was not relevant, but 22 percent of the ID agreed 
that the video was not relevant to the lesson content. The ED subjects seemed to look more intensely at the 
actual design and expressed some concerns including a more critical evaluation of the video portions of the 
program. 

The length of the video segments in relationship to the information given and the relevancy of 
specific visuals were critical points. Subjects expressed a desire to stop the video and replay only small 
sections rather than longer pieces. This suggests a need to make sure there is a good video match, but even 
further that shorter segments of video be used so that learners can repeat targeted sections. 

Navigation, Functionality 

Navigating through the software was a potentially problematic area for SW subjects who had little 
experience with this type of mediated instruction. Ninety percent of me SW subjects and 100 percent of the 
ID subjects agreed or strongly agreed that the module content was easy while no one thought it was difficult 
or hard to use. Despite that fact that no one thought is was difficult, 9 percent of the SW subjects and 22 
percent of the ID subjects agreed or strongly agreed that the content was confusing. 

Even though the design team advocates users choice, there are some controls on navigation to 
assist with directing the learners to prerequisite information or the desire of the subject matter expert (SME) 
to be sure that the learners cover a certain area within a segment. The ID subjects saw this as restrictive, 
but made the comment that this may be a good design feature for new users to help prevent them from 
getting lost within the program. The beta test revealed need to make sure all check marks stayed in place 
for the duration of a learner's use of a module. The check marks indicate topics and subtopics that have 
been completed. Learners wanted to be able to back track one screen at a time to recheck information rather 
than going back to the beginning of a section. 

The bookmark feature was designed as an easy way to restart the lesson after taking a break.. It 
kept information such as check marks to record progress and the specific place where a user left off during 
the last usage. Learners' individual records were stored on their own floppy disk upon exiting form the 
program. Among the information tracked is length of time spent on the module; that is important to 
enable learners to get continuing education credit. 

Although hot words with definitions in pop-up boxes, an on-line glossary, and content map to 
check where they were and where they could hop to were included, but they were very virtually unused. All 
subjects tended to go straight through the program in a linear sequence. The bookmark feature was 
something that might be used in a real work setting, but here each person only used it as part of the test. 
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The suggestions gathered led to changes in content, screen design, and general flow of the program. 
It will be important to give some sort of orientation to the next group so that they know how specific 
features can help them. To help clarify the lesson and alleviate confusion, revisions to the content 
introductions, transitions, areas of emphasis, and content map were made. 



The average length of time in minutes to complete the module for SW subjects was 138 (low SO, 
high 230, SD 48.95) and for ID subjects was 142 (low 90, high 180, SD 30.32). Approximately 80 
percent SW and 55 percent ID believed that the lesson was about the right length, while 9 percent SW and 
1 1 percent ID believed that it was boring. 

Strengths 

The overall reaction about the strengths of the lesson referred to the power of the video to help 
visualize concepts and the good organization and logical flow of content categories. The aural, visual, and 
textual interaction with the learner was viewed as an appropriate way to accommodate the different learning 
styles. 

The SW subjects were particularly pleased with the opportunity to see actual vignettes of 
situations that related directly to the content, both SW and ID subjects believed the navigation was 
friendly enough to help alleviate some potential confusion about certain areas of the content. 

Weaknesses 

The main weaknesses of the lesson concerned the relationship of review and test questions to 
content presented, the relevancy and length of video segments, and the inability to go back a few screens 
without taking a large jump backwards. 

Although the instructional designer had worked with the SME to write appropriate objectives and 
good questions matched to the objectives, the content presentation fell short on emphasizing the important 
points. In fact some questions contradicted the content and therefore were confusing. Revisions to the 
lesson presentation were necessary to provide enough clear and consistent information related to the test 
questions. 

Colors and Graphics 

Reactions to colors and graphics are most likely related to the learner's expenence. The SW 
subjects like the way that the graphics were presented and the colors that were used, while the ID subjects 
wanted the graphics to be more lively and found the colors dull. The two groups had opposite views here, 
reflecting their experience and expectations. 

CONCLUSIONS 

In general, the subjects responded favorably to the multimedia training and believed that the video 
component added value to the experience through the use of examples and demonstrations. A few S W 
subjects expressed the preference for traditional classroom instruction where they could ask questions. The 
more experienced social workers made suggestions to improve the content, while the less experienced 
students and instructional designers focused their comments more on the technology and program flow. 

Although the design team had built some fancy features into the module, the subjects did not use 
most of them. That could have been due to their limited experience with multimedia and technology in 
general, a way of presenting the features that is less than obvious, or lack of time to explore. 

The design adjustments recommended from the beta test can be categorized into three areas: 
content, instructional design, and technical integrity. With the subject matter expert being a member of the 
design team, the content and clarity was one of the positive features. In addition to attending to the features 
noted earlier, we recommend chunking the information better and adding even moie variety of perspectives 
to cover a diverse population of learners. We further recommend the inclusion of a person within the design 
team who could be classified as an expert who caii authenticate certain ideas from within a cultural 
perspective. In this case, the soc: J workers were very helpful as part of the beta test team. Experts on 
child development and special education were also included. 




The instructional design features mentioned earlier were corrected and the actual design interface 
was revised to improve navigation. Changes were made in fonts, colors, layout of the screen. One 
drawback of the SMEs being the project directors has been pressure to present information in a way that the 
designers felt matched linear video, but not necessarily multimedia. Those issues were revealed and 
addressed with each member of the design team contributing a strength. 

The decision was made to revise the program that records the learner's path throughout the 
program. The bookmark feature was modified and improved to allow more flexibility. It is also important 
to pinpoint a the specific features of hardware that the learners will have in their various remote sites 
> f^cause a few of the keyboard variations can make difference in program functionality. The programmer 
Liade changes to avoid sensitive features that can be anticipated, but this limits our ability to be specific 
with learner instructions. 

We recommend that the module be field tested in a variety of sites prior to full distribution. 
Within the field test we recommend that the data collected include measures of attitude, confidence, 
expectancy, comfort level with technology, knowledge, achievement, and satisfaction with the multimedia 
lesson. 
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